I haven't witnessed this issue firsthand so if Quicktime is 'locked up' and unresponsive you really don't have much choice I'm afraid once it is deadlocked there is little you can do to make it save work. If you are lucky the app may restore it's open files after a reboot but that is a risky option - it may not save the state if it has crashed.
If it isn't frozen or you can cancel the current task try the export option in the file menu. It should allow you to save to another disk if you choose an option like 1080p or an appropriate size. I suspect it still may want to save some data to the system disk, however it may work if you have an external disk to save onto instead of relying on the system disk alone.
You can also quit other apps in the hopes that they free up some disk space but I doubt that will help at all.
You could look at the default locations where Quicktime stores recordings, I think screen captures are saved in [HOME]/Library/Containers/com.apple.QuickTimePlayerX/Data/Library/Autosave Information, but I am using a different OS X version so that may have changed. You need to select this in Finder, "Go > hold 'alt' & click Library" to see the hidden Library folder contents. I don't know if other recordings are saved there too.
It may help if you describe how you created these files, it's possible they may exist somewhere out of site that you can recover later.